feat(scanner): Add a new `zebra-scanner` binary #8608

oxarbitrage · 2024-06-12T21:46:31Z

Motivation

We want to run the blockchain scanner in a new binary and process.

Close #8581
Close #8633

PR Author Checklist

Check before marking the PR as ready for review:

Will the PR name make sense to users?
Does the PR have a priority label?
Have you added or updated tests?
Is the documentation up to date?

Solution

Use zebra_state::init_read_only to run zebra_scan::spawn_init from a new binary.

Testing

Arguments:

     Running `target/debug/zebra-scanner --help`
zebra-scan 0.1.0-alpha.6
zebra-scanner arguments

USAGE:
    zebra-scanner [OPTIONS] --zebra-rpc-listen-addr <zebra-rpc-listen-addr>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
        --listen-addr <listen-addr>                         IP address and port for the gRPC server
        --network <network>                                 The Zcash network [default: Mainnet]
        --sapling-keys-to-scan <sapling-keys-to-scan>...    The sapling keys to scan for
        --scanning-cache-dir <scanning-cache-dir>           Path to scanning state [default: /home/alfredo/.cache/zebra]
        --zebra-rpc-listen-addr <zebra-rpc-listen-addr>
            The listen address of Zebra's RPC server used by the syncer to check for chain tip changes and get blocks in
            Zebra's non-finalized state
        --zebrad-cache-dir <zebrad-cache-dir>               Path to zebrad state [default: /home/alfredo/.cache/zebra]

I am using the following command to run the scanner with the ZECPAGES key, open the grpc port and see the logs:

RUST_LOG=info cargo run --bin zebra-scanner --features="shielded-scan" -- --sapling-keys-to-scan {\"key\":\"zxviews1q0duytgcqqqqpqre26wkl45gvwwwd706xw608hucmvfalr759ejwf7qshjf5r9aa7323zulvz6plhttp5mltqcgs9t039cx2d09mgq05ts63n8u35hyv6h9nc9ctqqtue2u7cer2mqegunuulq2luhq3ywjcz35yyljewa4mgkgjzyfwh6fr6jd0dzd44ghk0nxdv2hnv4j5nxfwv24rwdmgllhe0p8568sgqt9ckt02v2kxf5ahtql6s0ltjpkckw8gtymxtxuu9gcr0swvz\"} --zebrad-cache-dir "/media/alfredo/stuff/chain/zebra" --listen-addr "127.0.0.1:8231" --zebra-rpc-listen-addr "127.0.0.1:8232"

Output:

     Running `target/debug/zebra-scanner --sapling-keys-to-scan '{"key":"zxviews1q0duytgcqqqqpqre26wkl45gvwwwd706xw608hucmvfalr759ejwf7qshjf5r9aa7323zulvz6plhttp5mltqcgs9t039cx2d09mgq05ts63n8u35hyv6h9nc9ctqqtue2u7cer2mqegunuulq2luhq3ywjcz35yyljewa4mgkgjzyfwh6fr6jd0dzd44ghk0nxdv2hnv4j5nxfwv24rwdmgllhe0p8568sgqt9ckt02v2kxf5ahtql6s0ltjpkckw8gtymxtxuu9gcr0swvz"}' --zebrad-cache-dir /media/alfredo/stuff/chain/zebra --listen-addr '127.0.0.1:8231' --zebra-rpc-listen-addr '127.0.0.1:8232'`
2024-06-24T18:36:42.193678Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: trying to open current database format running_version=25.3.0
2024-06-24T18:36:42.193959Z  INFO zebra_state::service::finalized_state::disk_db: the open file limit is high enough for Zebra current_limit=1024 min_limit=512 ideal_limit=1024
2024-06-24T18:36:42.226420Z  INFO zebra_state::service::finalized_state::disk_db: creating secondary db directory create_dir_result=Ok(())
2024-06-24T18:36:46.533748Z  INFO zebra_state::service::finalized_state::disk_db: Opened Zebra state cache at /media/alfredo/stuff/chain/zebra/state/v25/mainnet
2024-06-24T18:36:46.534008Z  INFO zebra_state::service::finalized_state: loaded Zebra state cache tip=Some((Height(2547784), block::Hash("0000000000d74176f15e0f742a43b70723899a518d708c079d69b46662ca6a5d")))
2024-06-24T18:36:46.534745Z  INFO zebra_scan::init: starting scan service config=Config { sapling_keys_to_scan: 1, db_config: Config { cache_dir: "/home/alfredo/.cache/zebra", ephemeral: false, delete_old_database: true, debug_stop_at_height: None, debug_validity_check_interval: None } }
2024-06-24T18:36:46.534980Z  INFO zebra_state::service::finalized_state::disk_format::upgrade: creating new database with the current format running_version=1.0.0
2024-06-24T18:36:46.535066Z  INFO zebra_state::service::finalized_state::disk_db: the open file limit is high enough for Zebra current_limit=1024 min_limit=512 ideal_limit=1024
2024-06-24T18:36:46.552968Z  INFO zebra_state::service::finalized_state::disk_db: Opened Zebra state cache at /home/alfredo/.cache/zebra/private-scan/v1/mainnet
2024-06-24T18:36:46.553081Z  INFO zebra_scan::storage::db: loaded Zebra scanner cache
2024-06-24T18:36:46.553416Z  INFO zebra_scan::init: starting scan gRPC server listen_addr=127.0.0.1:8231
2024-06-24T18:36:46.553464Z  INFO zebra_scan::service::scan_task::scan: starting scan task network=Mainnet
2024-06-24T18:36:46.600463Z  INFO zebra_scan::service::scan_task::scan: scanner finished waiting for Sapling activation. Current tip: 2547784, Sapling activation: 419200
2024-06-24T18:36:46.600681Z  INFO zebra_scan::service::scan_task::scan: got min scan height start_height=Height(419199)
2024-06-24T18:37:08.772641Z  INFO zebra_scan::service::scan_task::scan: Scanning the blockchain for key 0, started at block 419200, now at block 420000, current tip 2547784
2024-06-24T18:38:40.466124Z  INFO zebra_scan::service::scan_task::scan: Scanning the blockchain for key 0, started at block 419200, now at block 430000, current tip 2547784
2024-06-24T18:40:08.461023Z  INFO zebra_scan::service::scan_task::scan: Scanning the blockchain for key 0, started at block 419200, now at block 440000, current tip 2547784
2024-06-24T18:41:38.833909Z  INFO zebra_scan::service::scan_task::scan: Scanning the blockchain for key 0, started at block 419200, now at block 450000, current tip 2547784

...

Review

Reviewer Checklist

Check before approving the PR:

Does the PR scope match the ticket?
Are there enough tests to make sure it works? Do the tests cover the PR motivation?
Are all the PR blockers dealt with?
PR blockers can be dealt with in new tickets or PRs.

And check the PR Author checklist is complete.

Follow Up Work

zebra-scan/src/bin/scanner/main.rs

oxarbitrage · 2024-06-12T22:12:51Z

Need changelog entry.

zebra-scan/src/bin/scanner/main.rs

arya2

This looks good, it may need to select the env-filter feature on tracing-subscriber, but otherwise I see no blockers.

Thank you!

zebra-scan/src/bin/scanner/main.rs

zebra-scan/Cargo.toml

oxarbitrage · 2024-06-17T19:37:47Z

This should be merged only after the next release. #8321

zebra-scan/src/bin/scanner/main.rs

arya2

This looks good!

The base branch was changed.

oxarbitrage added A-blockchain-scanner Area: Blockchain scanner of shielded transactions P-Medium ⚡ labels Jun 12, 2024

github-actions bot added the C-feature Category: New features label Jun 12, 2024

oxarbitrage commented Jun 12, 2024

View reviewed changes

zebra-scan/src/bin/scanner/main.rs Outdated Show resolved Hide resolved

zebra-scan/src/bin/scanner/main.rs Show resolved Hide resolved

oxarbitrage commented Jun 14, 2024

View reviewed changes

zebra-scan/src/bin/scanner/main.rs Outdated Show resolved Hide resolved

arya2 reviewed Jun 14, 2024

View reviewed changes

zebra-scan/src/bin/scanner/main.rs Outdated Show resolved Hide resolved

zebra-scan/Cargo.toml Outdated Show resolved Hide resolved

zebra-scan/Cargo.toml Outdated Show resolved Hide resolved

oxarbitrage added the do-not-merge Tells Mergify not to merge this PR label Jun 17, 2024

oxarbitrage mentioned this pull request Jun 20, 2024

Use sync::init_read_state_with_syncer() in the zebra-scanner #8633

Closed

2 tasks

arya2 reviewed Jun 20, 2024

View reviewed changes

zebra-scan/src/bin/scanner/main.rs Outdated Show resolved Hide resolved

oxarbitrage changed the base branch from main to non-final-read-state June 21, 2024 19:09

oxarbitrage force-pushed the zebra-scanner-binary branch from 1f056c6 to 3bb5a9e Compare June 21, 2024 23:00

oxarbitrage marked this pull request as ready for review June 24, 2024 18:46

oxarbitrage requested a review from a team as a code owner June 24, 2024 18:46

oxarbitrage requested review from arya2 and removed request for a team June 24, 2024 18:46

arya2 previously approved these changes Jun 27, 2024

View reviewed changes

oxarbitrage mentioned this pull request Jun 28, 2024

tests(scanner): Move zebra scanner tests to binary #8659

Merged

7 tasks

mpguerra linked an issue Jul 1, 2024 that may be closed by this pull request

Use sync::init_read_state_with_syncer() in the zebra-scanner #8633

Closed

2 tasks

arya2 removed the do-not-merge Tells Mergify not to merge this PR label Jul 2, 2024

Base automatically changed from non-final-read-state to main July 9, 2024 14:15

mergify bot requested a review from a team as a code owner July 9, 2024 14:15

mergify bot requested review from arya2 and removed request for a team July 9, 2024 14:15

oxarbitrage added 4 commits July 9, 2024 12:05

add a new zebra-scanner binary

f3e82ad

update arguments

cd2f488

allow birthday in config

171610f

remove required feature

a9bc25c

oxarbitrage and others added 3 commits July 9, 2024 12:06

add env-filter feature to tracing-subscriber dependency

8713575

use sync task

923ba84

codespell

23f72e9

oxarbitrage force-pushed the zebra-scanner-binary branch from 01fdf97 to 23f72e9 Compare July 9, 2024 15:14

oxarbitrage self-assigned this Jul 9, 2024

arya2 approved these changes Jul 9, 2024

View reviewed changes

mergify bot merged commit a94b2be into main Jul 9, 2024
132 checks passed

mergify bot deleted the zebra-scanner-binary branch July 9, 2024 18:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(scanner): Add a new `zebra-scanner` binary #8608

feat(scanner): Add a new `zebra-scanner` binary #8608

oxarbitrage commented Jun 12, 2024 •

edited

Loading

oxarbitrage commented Jun 12, 2024

arya2 left a comment

oxarbitrage commented Jun 17, 2024 •

edited

Loading

arya2 left a comment

feat(scanner): Add a new zebra-scanner binary #8608

feat(scanner): Add a new zebra-scanner binary #8608

Conversation

oxarbitrage commented Jun 12, 2024 • edited Loading

Motivation

PR Author Checklist

Check before marking the PR as ready for review:

Solution

Testing

Review

Reviewer Checklist

Follow Up Work

oxarbitrage commented Jun 12, 2024

arya2 left a comment

Choose a reason for hiding this comment

oxarbitrage commented Jun 17, 2024 • edited Loading

arya2 left a comment

Choose a reason for hiding this comment

feat(scanner): Add a new `zebra-scanner` binary #8608

feat(scanner): Add a new `zebra-scanner` binary #8608

oxarbitrage commented Jun 12, 2024 •

edited

Loading

oxarbitrage commented Jun 17, 2024 •

edited

Loading